Cognitive Dynamics Estimation: A whole-brain spatial regression paradigm for extracting the temporal dynamics of cognitive processes.

Functional MRI (fMRI) has been instrumental in understanding how cognitive processes are spatially mapped in the brain, yielding insights about brain regions and functions. However, in case the orthogonality of behavioral or stimulus timing is not guaranteed, the estimated brain maps fail to dissociate each cognitive process, and the resultant maps become unstable. Also, the brain mapping exercise can not provide temporal information on the cognitive process. Here we propose a qualitatively different approach to fMRI analysis, named Cognitive Dynamics Estimation (CDE), that estimates how multiple cognitive processes change over time even when behavior or stimulus logs are unavailable. This method transposes the conventional brain mapping; the brain activity pattern at each time point is subject to regression analysis with data-driven maps of cognitive processes as regressors, resulting in the time series of cognitive processes. The estimated time series captured the fluctuation of intensity and timing of cognitive processes on a trial-by-trial basis, which conventional analysis could not capture. Notably, the estimated time series predicted participants' cognitive ability to perform each psychological task. As an addition to our fMRI analytic toolkit, these results suggest the potential for CDE to elucidate underexplored cognitive phenomena, especially in the temporal domain.

Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans.

Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. Recently, neural decoding schemes, such as machine learning in functional magnetic resonance imaging (fMRI), has been employed to elucidate the underlying neural mechanisms. However, it remains unclear as to how spatially distributed brain regions temporally represent visual object categories and sub-categories. One promising strategy to address this issue is neural decoding with concurrently obtained neural response data of high spatial and temporal resolution. In this study, we explored the spatial and temporal organization of visual object representations using concurrent fMRI and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). We hypothesized that neural decoding by multimodal neural data with DNN would show high classification performance in visual object categorization (faces or non-face objects) and sub-categorization within faces and objects. Visualization of the fMRI DNN was more sensitive than that in the univariate approach and revealed that visual categorization occurred in brain-wide regions. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of the two DNNs improved the classification performance for both categorization and sub-categorization compared with fMRI DNN or EEG DNN alone. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner, and provide strong evidence of the ability of multimodal deep learning to uncover spatiotemporal neural machinery in sensory processing.

The Stroop effect involves an excitatory-inhibitory fronto-cerebellar loop.

The Stroop effect is a classical, well-known behavioral phenomenon in humans that refers to robust interference between language and color information. It remains unclear, however, when the interference occurs and how it is resolved in the brain. Here we show that the Stroop effect occurs during perception of color-word stimuli and involves a cross-hemispheric, excitatory-inhibitory loop functionally connecting the lateral prefrontal cortex and cerebellum. Participants performed a Stroop task and a non-verbal control task (which we term the Swimmy task), and made a response vocally or manually. The Stroop effect involved the lateral prefrontal cortex in the left hemisphere and the cerebellum in the right hemisphere, independently of the response type; such lateralization was absent during the Swimmy task, however. Moreover, the prefrontal cortex amplified cerebellar activity, whereas the cerebellum suppressed prefrontal activity. This fronto-cerebellar loop may implement language and cognitive systems that enable goal-directed behavior during perceptual conflicts.

Perceptual Uncertainty Alternates Top-down and Bottom-up Fronto-Temporal Network Signaling during Response Inhibition.

Response inhibition is a primary executive control function that allows the withholding of inappropriate responses, and requires appropriate perception of the external environment to achieve a behavioral goal. It remains unclear, however, how response inhibition is achieved when goal-relevant information involves perceptual uncertainty. Twenty-six human participants of both sexes performed a go/no-go task where visually presented random-dot motion stimuli involved perceptual uncertainties. The right inferior frontal cortex (rIFC) was involved in response inhibition, and the middle temporal (MT) region showed greater activity when dot motions involved less uncertainty. A neocortical temporal region in the superior temporal sulcus (STS) specifically showed greater activity during response inhibition in more perceptually certain trials. In this STS region, activity was greater when response inhibition was successful than when it failed. Directional effective connectivity analysis revealed that, in more coherent trials, the MT and STS regions showed enhanced connectivity to the rIFC, whereas in less coherent trials, the signal direction was reversed. These results suggest that a reversible fronto-temporal functional network guides response inhibition and perceptual decision-making under perceptual uncertainty, and in this network, perceptual information in the MT is converted to control information in the rIFC via STS, enabling achievement of response inhibition.SIGNIFICANCE STATEMENT Response inhibition refers to withholding inappropriate behavior and is important for achieving goals. Often, however, decision must be made based on limited environmental evidence. We showed that successful response inhibition is guided by a neocortical temporal region that plays a hub role in converting perceived information coded in a posterior temporal region to control information coded in the PFC. Interestingly, when a perceived stimulus becomes more uncertain, the PFC supplements stimulus encoding in the temporal regions. Our results highlight fronto-temporal mechanisms of response inhibition in which conversion of stimulus-control information is regulated based on the uncertainty of environmental evidence.

On co-activation pattern analysis and non-stationarity of resting brain activity.

The non-stationarity of resting-state brain activity has received increasing attention in recent years. Functional connectivity (FC) analysis with short sliding windows and coactivation pattern (CAP) analysis are two widely used methods for assessing the dynamic characteristics of brain activity observed with functional magnetic resonance imaging (fMRI). However, the statistical nature of the dynamics captured by these techniques needs to be verified. In this study, we found that the results of CAP analysis were similar for real fMRI data and simulated stationary data with matching covariance structures and spectral contents. We also found that, for both the real and simulated data, CAPs were clustered into spatially heterogeneous modules. Moreover, for each of the modules in the real data, a spatially similar module was found in the simulated data. The present results suggest that care needs to be taken when interpreting observations drawn from CAP analysis as it does not necessarily reflect non-stationarity or a mixture of states in resting brain activity.

Executive control by fronto-parietal activity explains counterintuitive decision behavior in complex value-based decision-making.

In real life, humans make decisions by taking into account multiple independent factors, such as delay and probability. Cognitive psychology suggests that cognitive control mechanisms play a key role when facing such complex task conditions. However, in value-based decision-making, it still remains unclear to what extent cognitive control mechanisms become essential when the task condition is complex. In this study, we investigated decision-making behaviors and underlying neural mechanisms using a multifactor gambling task where participants simultaneously considered probability and delay. Decision-making behavior in the multifactor task was modulated by both probability and delay. The behavioral effect of probability was stronger than delay, consistent with previous studies. Furthermore, in a subset of conditions that recruited fronto-parietal activations, reaction times were paradoxically elongated despite lower probabilistic uncertainty. Notably, such a reaction time elongation did not occur in control tasks involving single factors. Meta-analysis of brain activations suggested an interpretation that the paradoxical increase of reaction time may be associated with strategy switching. Consistent with this interpretation, logistic regression analysis of the behavioral data suggested a presence of multiple decision strategies. Taken together, we found that a novel complex value-based decision-making task cause prominent activations in fronto-parietal cortex. Furthermore, we propose that these activations can be interpreted as recruitment of cognitive control system in complex situations.

Reversible Fronto-occipitotemporal Signaling Complements Task Encoding and Switching under Ambiguous Cues.

Adaptation to changing environments involves the appropriate extraction of environmental information to achieve a behavioral goal. It remains unclear how behavioral flexibility is guided under situations where the relevant behavior is ambiguous. Using functional brain mapping of machine learning decoders and directional functional connectivity, we show that brain-wide reversible neural signaling underpins task encoding and behavioral flexibility in ambiguously changing environments. When relevant behavior is cued ambiguously during behavioral shifting, neural coding is attenuated in distributed cortical regions, but top-down signals from the prefrontal cortex complement the coding. When behavioral shifting is cued more explicitly, modality-specialized occipitotemporal regions implement distinct neural coding about relevant behavior, and bottom-up signals from the occipitotemporal region to the prefrontal cortex supplement the behavioral shift. These results suggest that our adaptation to an ever-changing world is orchestrated by the alternation of top-down and bottom-up signaling in the fronto-occipitotemporal circuit depending on the availability of environmental information.

A method for reconstruction of interpretable brain networks from transient synchronization in resting-state BOLD fluctuations.

Resting-state (rs) fMRI has been widely used to examine brain-wide large-scale spatiotemporal architectures, known as resting-state networks (RSNs). Recent studies have focused on the temporally evolving characteristics of RSNs, but it is unclear what temporal characteristics are reflected in the networks. To address this issue, we devised a novel method for voxel-based visualization of spatiotemporal characteristics of rs-fMRI with a time scale of tens of seconds. We first extracted clusters of dominant activity-patterns using a region-of-interest approach and then used these temporal patterns of the clusters to obtain voxel-based activation patterns related to the clusters. We found that activation patterns related to the clusters temporally evolved with a characteristic temporal structure and showed mutual temporal alternations over minutes. The voxel-based representation allowed the decoding of activation patterns of the clusters in rs-fMRI using a meta-analysis of functional activations. The activation patterns of the clusters were correlated with behavioral measures. Taken together, our analysis highlights a novel approach to examine brain activity dynamics during rest.

Prefrontal-Striatal Mechanisms of Behavioral Impulsivity During Consumption of Delayed Real Liquid Rewards.

Intertemporal choice involves the evaluation of future rewards and reflects behavioral impulsivity. After choosing a delayed reward in an intertemporal choice, a behavioral agent waits for, receives, and then consumes the reward. The current study focused on the consumption of the delayed reward and examined the neural mechanisms of behavioral impulsivity. In humans consuming delayed real liquid rewards in an intertemporal choice, the ventral striatum (VS) showed differential activity between anterior (aVS) and posterior (pVS) regions depending on the degree of behavioral impulsivity. Additionally, impulsive individuals showed activity in the anterior prefrontal cortex (aPFC). An analysis of task-related effective connectivity based on psychophysiological interaction (PPI) revealed that PPI was robust from the aPFC to pVS, but not in the opposite direction. On the other hand, strong bidirectional PPIs were observed between the aVS and pVS, but PPIs from the pVS to aVS were enhanced in impulsive individuals. These results suggest that behavioral impulsivity is reflected in aPFC-VS mechanisms during the consumption of delayed real liquid rewards.

Cross-Hemispheric Complementary Prefrontal Mechanisms during Task Switching under Perceptual Uncertainty.

Flexible adaptation to changing environments is a representative executive control function implicated in the frontoparietal network that requires appropriate extraction of goal-relevant information through perception of the external environment. It remains unclear, however, how the flexibility is achieved under situations where goal-relevant information is uncertain. To address this issue, the current study examined neural mechanisms for task switching in which task-relevant information involved perceptual uncertainty. Twenty-eight human participants of both sexes alternated behavioral tasks in which they judged motion direction or color of visually presented colored dot stimuli that moved randomly. Task switching was associated with frontoparietal regions in the left hemisphere, and perception of ambiguous stimuli involved contralateral homologous frontoparietal regions. On the other hand, in stimulus-modality-dependent occipitotemporal regions, task coding information was increased during task switching. Effective connectivity analysis revealed that the frontal regions signaled toward the modality-dependent occipitotemporal regions when a relevant stimulus was more ambiguous, whereas the occipitotemporal regions signaled toward the frontal regions when the stimulus was more distinctive. These results suggest that complementary prefrontal mechanisms in the left and right hemispheres help to achieve a behavioral goal when the external environment involves perceptual uncertainty.SIGNIFICANCE STATEMENT In our daily life, environmental information to achieve a goal is not always certain, but we make judgments in such situations, and change our behavior accordingly. This study examined how the flexibility of behavior is achieved in a situation where goal-relevant information involves perceptual uncertainty. fMRI revealed that the lateral prefrontal cortex (PFC) in the left hemisphere is associated with behavioral flexibility, and the perception of ambiguous stimuli involves the PFC in the right hemisphere. These bilateral PFC signaled to stimulus-modality-dependent occipitotemporal regions, depending on perceptual uncertainty and the task to be performed. These top-down signals supplement task coding in the occipitotemporal regions, and highlight interhemispheric prefrontal mechanisms involved in executive control and perceptual decision-making.

Counterfactual Explanation of Brain Activity Classifiers Using Image-To-Image Transfer by Generative Adversarial Network.

Deep neural networks (DNNs) can accurately decode task-related information from brain activations. However, because of the non-linearity of DNNs, it is generally difficult to explain how and why they assign certain behavioral tasks to given brain activations, either correctly or incorrectly. One of the promising approaches for explaining such a black-box system is counterfactual explanation. In this framework, the behavior of a black-box system is explained by comparing real data and realistic synthetic data that are specifically generated such that the black-box system outputs an unreal outcome. The explanation of the system's decision can be explained by directly comparing the real and synthetic data. Recently, by taking advantage of advances in DNN-based image-to-image translation, several studies successfully applied counterfactual explanation to image domains. In principle, the same approach could be used in functional magnetic resonance imaging (fMRI) data. Because fMRI datasets often contain multiple classes (e.g., multiple behavioral tasks), the image-to-image transformation applicable to counterfactual explanation needs to learn mapping among multiple classes simultaneously. Recently, a new generative neural network (StarGAN) that enables image-to-image transformation among multiple classes has been developed. By adapting StarGAN with some modifications, here, we introduce a novel generative DNN (counterfactual activation generator, CAG) that can provide counterfactual explanations for DNN-based classifiers of brain activations. Importantly, CAG can simultaneously handle image transformation among all the seven classes in a publicly available fMRI dataset. Thus, CAG could provide a counterfactual explanation of DNN-based multiclass classifiers of brain activations. Furthermore, iterative applications of CAG were able to enhance and extract subtle spatial brain activity patterns that affected the classifier's decisions. Together, these results demonstrate that the counterfactual explanation based on image-to-image transformation would be a promising approach to understand and extend the current application of DNNs in fMRI analyses.

Self-Controlled Choice Arises from Dynamic Prefrontal Signals That Enable Future Anticipation.

Self-control allows humans the patience necessary to maximize reward attainment in the future. Yet it remains elusive when and how the preference to self-controlled choice is formed. We measured brain activity while female and male humans performed an intertemporal choice task in which they first received delayed real liquid rewards (forced-choice trial), and then made a choice between the reward options based on the experiences (free-choice trial). We found that, while subjects were awaiting an upcoming reward in the forced-choice trial, the anterior prefrontal cortex (aPFC) tracked a dynamic signal reflecting the pleasure of anticipating the future reward. Importantly, this prefrontal signal was specifically observed in self-controlled individuals, and moreover, interregional negative coupling between the prefrontal region and the ventral striatum (VS) became stronger in those individuals. During consumption of the liquid rewards, reduced ventral striatal activity predicted self-controlled choices in the subsequent free-choice trials. These results suggest that a well-coordinated prefrontal-striatal mechanism during the reward experience shapes preferences regarding the future self-controlled choice.SIGNIFICANCE STATEMENT Anticipating future desirable events is a critical mental function that guides self-controlled behavior in humans. When and how are the self-controlled choices formed in the brain? We monitored brain activity while humans awaited a real liquid reward that became available in tens of seconds. We found that the frontal polar cortex tracked temporally evolving signals reflecting the pleasure of anticipating the future reward, which was enhanced in self-controlled individuals. Our results highlight the contribution of the fronto-polar cortex to the formation of self-controlled preferences, and further suggest that future prospect in the prefrontal cortex (PFC) plays an important role in shaping future choice behavior.

Within-Subject Correlation Analysis to Detect Functional Areas Associated With Response Inhibition.

Functional areas in fMRI studies are often detected by brain-behavior correlation, calculating across-subject correlation between the behavioral index and the brain activity related to a function of interest. Within-subject correlation analysis is also employed in a single subject level, which utilizes cognitive fluctuations in a shorter time period by correlating the behavioral index with the brain activity across trials. In the present study, the within-subject analysis was applied to the stop-signal task, a standard task to probe response inhibition, where efficiency of response inhibition can be evaluated by the stop-signal reaction time (SSRT). Since the SSRT is estimated, by definition, not in a trial basis but from pooled trials, the correlation across runs was calculated between the SSRT and the brain activity related to response inhibition. The within-subject correlation revealed negative correlations in the anterior cingulate cortex and the cerebellum. Moreover, the dissociation pattern was observed in the within-subject analysis when earlier vs. later parts of the runs were analyzed: negative correlation was dominant in earlier runs, whereas positive correlation was dominant in later runs. Regions of interest analyses revealed that the negative correlation in the anterior cingulate cortex, but not in the cerebellum, was dominant in earlier runs, suggesting multiple mechanisms associated with inhibitory processes that fluctuate on a run-by-run basis. These results indicate that the within-subject analysis compliments the across-subject analysis by highlighting different aspects of cognitive/affective processes related to response inhibition.

Large-scale network integration in the human brain tracks temporal fluctuations in memory encoding performance.

Although activation/deactivation of specific brain regions has been shown to be predictive of successful memory encoding, the relationship between time-varying large-scale brain networks and fluctuations of memory encoding performance remains unclear. Here, we investigated time-varying functional connectivity patterns across the human brain in periods of 30-40 s, which have recently been implicated in various cognitive functions. During functional magnetic resonance imaging, participants performed a memory encoding task, and their performance was assessed with a subsequent surprise memory test. A graph analysis of functional connectivity patterns revealed that increased integration of the subcortical, default-mode, salience, and visual subnetworks with other subnetworks is a hallmark of successful memory encoding. Moreover, multivariate analysis using the graph metrics of integration reliably classified the brain network states into the period of high (vs. low) memory encoding performance. Our findings suggest that a diverse set of brain systems dynamically interact to support successful memory encoding.

Visual perceptual training reconfigures post-task resting-state functional connectivity with a feature-representation region.

The neural mechanisms underlying visual perceptual learning (VPL) have typically been studied by examining changes in task-related brain activation after training. However, the relationship between post-task "offline" processes and VPL remains unclear. The present study examined this question by obtaining resting-state functional magnetic resonance imaging (fMRI) scans of human brains before and after a task-fMRI session involving visual perceptual training. During the task-fMRI session, participants performed a motion coherence discrimination task in which they judged the direction of moving dots with a coherence level that varied between trials (20, 40, and 80%). We found that stimulus-induced activation increased with motion coherence in the middle temporal cortex (MT+), a feature-specific region representing visual motion. On the other hand, stimulus-induced activation decreased with motion coherence in the dorsal anterior cingulate cortex (dACC) and bilateral insula, regions involved in decision making under perceptual ambiguity. Moreover, by comparing pre-task and post-task rest periods, we revealed that resting-state functional connectivity (rs-FC) with the MT+ was significantly increased after training in widespread cortical regions including the bilateral sensorimotor and temporal cortices. In contrast, rs-FC with the MT+ was significantly decreased in subcortical regions including the thalamus and putamen. Importantly, the training-induced change in rs-FC was observed only with the MT+, but not with the dACC or insula. Thus, our findings suggest that perceptual training induces plastic changes in offline functional connectivity specifically in brain regions representing the trained visual feature, emphasising the distinct roles of feature-representation regions and decision-related regions in VPL.

Intertemporal Decision-Making Involves Prefrontal Control Mechanisms Associated with Working Memory.

Intertemporal decision-making involves simultaneous evaluation of both the magnitude and delay to reward, which may require the integrated representation and comparison of these dimensions within working memory (WM). In the current study, neural activation associated with intertemporal decision-making was directly compared with WM load-related activation. During functional magnetic resonance imaging, participants performed an intermixed series of WM trials and intertemporal decision-making trials both varying in load, with the latter in terms of choice difficulty, via options tailored to each participant's subjective value function for delayed rewards. The right anterior prefrontal cortex (aPFC) and dorsolateral prefrontal cortex (dlPFC) showed activity modulation by choice difficulty within WM-related brain regions. In aPFC, these 2 effects (WM, choice difficulty) correlated across individuals. In dlPFC, activation increased with choice difficulty primarily in patient (self-controlled) individuals, and moreover was strongest when the delayed reward was chosen on the most difficult trials. Finally, the choice-difficulty effects in dlPFC and aPFC were correlated across individuals, suggesting a functional relationship between the 2 regions. Together, these results suggest a more precise account of the relationship between WM and intertemporal decision-making that is specifically tied to choice difficulty, and involves the coordinated activation of a lateral PFC circuit supporting successful self-control.

Data for behavioral results and brain regions showing a time effect during pair-association retrieval.

The current data article provides behavioral and neuroimaging data for the research article "Relatedness-dependent rapid development of brain activity in anterior temporal cortex during pair-association retrieval" (Jimura et al., 2016) [1]. Behavioral performance is provided in a table. Fig. 2 of the article is based on this table. Brain regions showing time effect are provided in a table. A statistical activation map for the time effect is shown in Fig. 3C of the article.

Relatedness-dependent rapid development of brain activity in anterior temporal cortex during pair-association retrieval.

Functional MRI studies have revealed that the brain activity in the anterior temporal cortex during memory retrieval increases over months after memory encoding. Behavioral evidence has demonstrated that long-term memory can sometimes be consolidated more rapidly in one or two days. In the present functional MRI study, we manipulated the relatedness between paired faces to be retrieved in a pair-association task. The brain activity in the anterior temporal cortex during retrieval of paired associates increased rapidly in one day, as shown in previous studies. We found that the speed of the brain activity development was dependent on the level of semantic relatedness of paired faces. The results suggest that the semantic relatedness enhances the speed of formation of memory representation in the anterior temporal cortex.

Dynamically Allocated Hub in Task-Evoked Network Predicts the Vulnerable Prefrontal Locus for Contextual Memory Retrieval in Macaques.

Neuroimaging and neurophysiology have revealed that multiple areas in the prefrontal cortex (PFC) are activated in a specific memory task, but severity of impairment after PFC lesions is largely different depending on which activated area is damaged. The critical relationship between lesion sites and impairments has not yet been given a clear mechanistic explanation. Although recent works proposed that a whole-brain network contains hubs that play integrative roles in cortical information processing, this framework relying on an anatomy-based structural network cannot account for the vulnerable locus for a specific task, lesioning of which would bring impairment. Here, we hypothesized that (i) activated PFC areas dynamically form an ordered network centered at a task-specific "functional hub" and (ii) the lesion-effective site corresponds to the "functional hub," but not to a task-invariant "structural hub." To test these hypotheses, we conducted functional magnetic resonance imaging experiments in macaques performing a temporal contextual memory task. We found that the activated areas formed a hierarchical hub-centric network based on task-evoked directed connectivity, differently from the anatomical network reflecting axonal projection patterns. Using a novel simulated-lesion method based on support vector machine, we estimated severity of impairment after lesioning of each area, which accorded well with a known dissociation in contextual memory impairment in macaques (impairment after lesioning in area 9/46d, but not in area 8Ad). The predicted severity of impairment was proportional to the network "hubness" of the virtually lesioned area in the task-evoked directed connectivity network, rather than in the anatomical network known from tracer studies. Our results suggest that PFC areas dynamically and cooperatively shape a functional hub-centric network to reallocate the lesion-effective site depending on the cognitive processes, apart from static anatomical hubs. These findings will be a foundation for precise prediction of behavioral impacts of damage or surgical intervention in human brains.

Enhancement of visual attention precedes the emergence of novel metaphor interpretations.

Emergence of creative ideas often involves sudden linking of unrelated ideas. Here we demonstrate that the production of insights about novel metaphor was preceded by prolonged visual inspection of metaphorical sentences. The participants made free spoken responses in a task requiring the generation of semantic interpretations about visually presented sentences, where gaze locations were continuously monitored. We found that creative interpretations were primarily generated from novel metaphorical expressions, rather than from conventional expressions. Moreover, presented words within novel metaphors were visually inspected for longer periods specifically before creative interpretations were produced. Interestingly, prolonged gazes occurred several seconds (∼8 s) prior to the generation of creative interpretations, particularly, in the case of the topic word within the metaphor. These results demonstrate latent cognitive process meditating the emergence of insights, and suggest that the prior visual inspection prompted the semantic representation of the metaphorical sentences, which eventually facilitates creative ideas.